Skip to content

fix(cache): record audit scope and reject mismatched cache reads (#542) - #549

Open
Marc-cn wants to merge 2 commits into
mainfrom
fix/542-cache-scope
Open

Marc-cn wants to merge 2 commits into
mainfrom
fix/542-cache-scope

Conversation

@Marc-cn

@Marc-cn Marc-cn commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

darnit audit --tags level=1 wrote a cache that remediation then read as a full audit (#542). Two rules fix it:

  1. Only an unfiltered audit is cached. A run narrowed by --tags, --include, --exclude or a profile does not write the audit cache. The CLI decides "filtered" by comparing the control count after filtering with the count loaded, so any future flag is covered.
  2. Remediation reads and invalidates through the same store and key. read_full_audit_cache() and invalidate_full_audit_cache() share the store resolution, so an operator's [stores.cache] is honoured on both sides. Before, invalidation always hit the default store and a configured store kept the stale audit.

The cache envelope records framework and level; CACHE_VERSION is 2, so caches from earlier versions are ignored and rebuilt.

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed (section 10.4)
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

  • Tests pass locally (uv run pytest tests/ -v): 5173 passed, 19 skipped

  • Added tests for new functionality (if applicable)

  • Linting passes (uv run ruff check .)

  • tests/darnit/core/test_audit_cache.py: the filtered decision, one key per repository, read hit and miss cases, invalidation through the default store, and invalidation through a configured local-fs store. The last one shows the old invalidate_audit_cache(path) leaving the configured entry in place and the new helper expiring it.

  • tests/darnit_baseline/test_audit_cache_scope.py: runs the real audit_openssf_baseline() with and without a profile, and checks read_full_audit_cache() is None only for the profile run.

  • By hand, on a temp repository: audit -t level=1 MISS, audit --include X MISS, full audit HIT (66 results), audit -t level=1 after the full audit still HIT with 66.

AI assistance

  • No AI assistance was used
  • AI assistance was used

Claude Fable 5.1 wrote the changes and drafted this description. I ran every command and checked the results myself. The commits carry an Assisted-by: Claude:claude-fable-5-1 trailer.

Additional Notes

darnit audit --profile (#576) is covered by the control-count rule here, so #576 needs no cache code of its own.

@Marc-cn
Marc-cn requested a review from mlieberman85 as a code owner October 3, 2026 23:09

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice start. A few gaps before this fully closes #542:

  1. Remediation reads without expected_level, but on a miss it re-audits at level 3, so a level-1 cache is still a hit. Please pass expected_level=3.
  2. --include/--exclude and profile= also produce filtered audits that currently look unfiltered. Please record the full scope (a filtered flag or the control-ID set).
  3. A filtered audit still replaces an unfiltered cache, and the remediation read ignores [stores.cache]. Fix both here, or change "Fixes #542" to "Refs".
  4. Bump CACHE_VERSION, or treat a missing tags key as a miss, so old cache files aren't read as unfiltered.
  5. Spec-first: describe the envelope change in framework-design.md 10.4.
  6. Add the model version to Assisted-by and name the tool in the PR body.

Drafted with Claude Code; reviewed and posted by me.

@Marc-cn
Marc-cn force-pushed the fix/542-cache-scope branch from 16fa6e3 to 56b39cc Compare October 6, 2026 22:33
@Marc-cn

Marc-cn commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed.

  1. expected_level=3 passed. _run_baseline_checks defaults to level=3, so accepting a level-1 cache would have remediated against a narrower audit than the one it would otherwise run
  2. Added a filtered boolean to the envelope rather than the control-ID set, --include/--exclude can't be expressed as a tag string, and a set would make the envelope large and the comparison brittle. run_sieve_audit has no profile parameter (that filtering happens in the MCP tool), so profile isn't covered here
  3. Both done. A filtered audit now writes under its own cache key, so it can't replace the unfiltered entry but repeat filtered runs still hit. Remediation resolves [stores.cache] via _load_merged_stores + resolve_stores and reproduces the same key derivation as tools/audit.py. Kept "Fixes Remediation trusts the audit cache regardless of framework, level, or tag filter #542".
  4. CACHE_VERSION 1 → 2, and the check is now != CACHE_VERSION rather than > — the old upper-bound form would still have read a v1 envelope, which has no tags/filtered keys and would present a possibly filtered audit as unfiltered.
  5. framework-design.md 10.4 gains two requirements: the envelope's scope fields plus the distinct key for filtered runs, and the scoped-read semantics including the version rule. The remediation requirement now also states the store it reads through.
  6. Trailer is `Assisted-by: Claude:claude-opus-4-5.

Tests: 38 in test_audit_cache.py (7 new across two classes), tests/darnit_baseline 900 passed, tests/darnit minus parity 2053 passed.

Three failures in the long tests/darnit run — test_mcp_equals_direct_equals_cli, TestProgressLines::test_progress_lines_use_n_over_m_counter, TestSecretRedaction::test_leaked_exception_message_does_not_reach_report, all pass individually and fail only under a full-suite run. I checked test_mcp_equals_direct_equals_cli against the unmodified lockfile and it fails identically there, so they're pre-existing. Probably worth their own issue.

I also left ruff format alone on cli.py and orchestrator.py: running it there reformats ~180 lines I didn't touch (171 files in the repo currently fail format --check), which seemed like a separate PR.

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this covers everything I asked. Two things before it can merge:

  1. Rebase onto main. It conflicts in orchestrator.py, cli.py and audit.py, so CI hasn't run. #552 changed _load_merged_stores to (framework_name, operator). After the rebase, _load_merged_stores(local_path, "openssf-baseline") would raise inside the try, and every remediation would quietly re-run the audit. Please resolve the operator configuration the way _run_baseline_checks does. Ideally, also move the cache-key derivation into one helper that both tools/audit.py and the orchestrator call, so the two can't drift apart again.
  2. Profile-filtered audits. audit_openssf_baseline(profile=...) narrows controls itself and passes tags=None, so its cache entry still says filtered=False, and remediation reads it as a full audit. Please fail closed: when a caller passes controls without saying cache_filtered, treat the audit as filtered. Callers that pass the full set (the MCP tool without a profile, the CLI without filters) then pass cache_filtered=False explicitly. Please add a test for the profile case.

On the three tests that fail in a full-suite run: main now passes the full suite (5210 passed after #548 made the gh calls offline). If they still fail after the rebase, please open an issue with the output.

Drafted with Claude Code; reviewed and posted by me.

@Marc-cn
Marc-cn force-pushed the fix/542-cache-scope branch from 56b39cc to 913771a Compare October 10, 2026 23:32
@Marc-cn

Marc-cn commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Rebuilt on current main and force-pushed (one commit). Main had rewritten the three conflicting files, so I re-applied the change instead of resolving the conflicts in place.

  1. Store loading and the cache key. Remediation now reads through read_full_audit_cache() in tools/audit.py. It resolves the operator configuration the way run_sieve_audit does, calls _load_merged_stores(framework_name, operator) with the new signature, and closes the bundle. The key comes from one helper, audit_cache_key(), used by the writer and the reader.
  2. Profile-filtered audits fail closed. run_sieve_audit records an audit as filtered when it has tags, when the caller says so, or when the caller passed controls without saying. The Baseline tool passes cache_filtered=bool(profile), the CLI computes it from --tags/--include/--exclude, and the generic MCP audit tool states cache_filtered=False. Profile test: tests/darnit_baseline/test_audit_cache_scope.py.

Checked through the CLI on a committed one-file repository:

Sequence Remediation read
main: audit -t level=1 HIT, 26 results, as if it were a full audit
this branch: audit -t level=1 MISS
this branch: audit --include OSPS-QA-05.01 MISS
this branch: full audit HIT, 66 results
this branch: full audit, then -t level=1 HIT, still 66 results
this branch, reading the cache main wrote MISS (old envelope version)

One existing test changed: test_run_sieve_audit_without_cache_write passes its own controls, so it now states cache_filtered=False.

Full suite: 5167 passed, 19 skipped. The three tests that failed for me before did not fail.

Note for #576: it wires --profile into darnit audit. Whichever of the two merges second should add args.profile to the CLI's cache_filtered.

Drafted with Claude; reviewed and posted by me.

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the rebuild covers both round-2 asks, and the W-03 probe now re-audits instead of saying "all controls are passing". A few things before merge:

  1. Remediation now reads through [stores.cache] but still invalidates the default store (orchestrator.py:1289, darnit_baseline/tools.py:512). With a configured store and a dirty tree, create_security_policy writes SECURITY.md and the next remediation still reads the old OSPS-VM-02.01 FAIL. Please invalidate through the same store/key as the read (a helper beside read_full_audit_cache), and update the invalidation requirement in framework-design.md 10.4 (line ~1899).
  2. Please refresh the PR body: it still says "partially" and lists store routing and filtered-run overwrite as not included, and it should name the AI tool used.
  3. cli.py:341 lists the filter flags by hand, so #576's --profile would be cached as unfiltered. Could the CLI derive cache_filtered from whether the filtered control list is smaller than the loaded one? That survives future flags.
  4. Smaller: nothing reads audit-cache-filtered, so not writing filtered audits at all would be simpler, and tags/expected_tags would then be redundant with filtered. The profile test could run the real audit_openssf_baseline(profile=...) and assert read_full_audit_cache(...) is None instead of patching run_sieve_audit.

Drafted with Claude Code; reviewed and posted by me.

A filtered audit left a cache that remediation read as a full audit. On
main, `darnit audit --tags level=1` writes 26 results and
remediate_audit_findings then acts on those as if all 66 controls had run.

The cache envelope now records the audit's scope: framework, level, tag
filter, and whether any filter narrowed the control set. CACHE_VERSION is 2
and must match exactly, so older envelopes, which cannot describe their own
scope, are ignored and rebuilt.

A filtered audit is written under its own key, so it never replaces the
full-audit entry. run_sieve_audit records an audit as filtered when it has
tags, when the caller says so, or when the caller passed its own `controls`
without saying. That last case fails closed: the function cannot tell a
narrowed list from the full set. Callers with the full set say
cache_filtered=False (the generic MCP audit tool); the CLI and the Baseline
tool compute it from --tags/--include/--exclude and from the profile.

Remediation reads through read_full_audit_cache, which resolves the
operator configuration and the store the way run_sieve_audit does, derives
the key with the same helper (audit_cache_key), and accepts only an
unfiltered level 3 audit of its framework.

Rebuilt on current main rather than rebased: main had rewritten the three
conflicting files since the original commit.

Drafted with Claude, edited by me.

Signed-off-by: Marc-cn <130138935+Marc-cn@users.noreply.github.com>
Assisted-by: Claude:claude-fable-5-1
…igured store

Addresses the review of Oct 12 on this PR.

- A filtered audit is not written to the cache at all. That replaces the
  separate "-filtered" key and the tags/filtered envelope fields, which
  nothing read. The read checks framework and level only.
- invalidate_full_audit_cache() resolves the same store and key as
  read_full_audit_cache(). Remediation used the default store, so with
  a configured [stores.cache] the old audit stayed in place.
- The CLI derives cache_filtered from the control count: fewer controls
  than were loaded means filtered. That covers --profile (#576) and any
  future flag without listing them.
- The profile test runs the real audit_openssf_baseline() and checks
  that a profile run leaves no full-audit cache.
- Spec 10.4 updated for the invalidation and the envelope.

Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Marc-cn <130138935+Marc-cn@users.noreply.github.com>
@Marc-cn
Marc-cn force-pushed the fix/542-cache-scope branch from 913771a to 732c143 Compare October 13, 2026 19:30
@Marc-cn

Marc-cn commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

@mlieberman85 the Oct 12 points, rebased onto main:

  1. Configured store: invalidate_full_audit_cache() resolves the same store and key as the read, and both call sites use it. New test writes to a local-fs store, shows the old call leaving it in place, and the new one expiring it. Spec 10.4 updated.
  2. PR body: rewritten, with the AI tool named.
  3. cache_filtered: the CLI now compares the control count after filtering with the loaded count. --profile from fix(cli): make darnit audit --profile filter the controls #576 is covered without changes there.
  4. Simpler: filtered audits are not written at all. The -filtered key and the tags/filtered fields are gone. The profile test runs the real audit_openssf_baseline() and asserts read_full_audit_cache() is None.

5173 tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants